VTC: Improving Video-Text Retrieval with User Comments

AbstractMulti-modal retrieval is an important problem for many applications, such as recommendation and search. Current benchmarks even datasets are often manually constructed consist of mostly clean samples where all modalities well-correlated with the content. Thus, current video-text literature largely focuses on video titles or audio transcripts, while ignoring user comments, since users tend to discuss topics only vaguely related video. Despite ubiquity comments online, there currently no multi-modal representation learning that includes comments. In this paper, we a) introduce a new dataset videos, comments; b) present attention-based mechanism allows model learn from sometimes irrelevant data c) show by using our method able better, more contextualised, representations image, representations. Project page: https://unitaryai.github.io/vtc-paper.